{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Spark Streaming\n",
    "\n",
    "_____\n",
    "### Note on  Streaming\n",
    "Streaming is something that is rapidly advancing and changin fast, there are multipl enew libraries every year, new and different services always popping up, and what is in this notebook may or may not apply to you. Maybe your looking for something specific on Kafka, or maybe you are looking for streaming about twitter, in which case Spark might be overkill for what you really want. Realistically speaking each situation is going to require a customized solution and this course is never going to be able to supply a one size fits all solution. Because of this, I wanted to point out some great resources for Python and Spark StreamingL\n",
    "\n",
    "* [The Official Documentation is great. This should be your first go to.](http://spark.apache.org/docs/latest/streaming-programming-guide.html#spark-streaming-programming-guide)\n",
    "\n",
    "* [Fantastic Guide to Spark Streaming with Kafka](https://www.rittmanmead.com/blog/2017/01/getting-started-with-spark-streaming-with-python-and-kafka/)\n",
    "\n",
    "* [Another Spark Streaming Example with Geo Plotting](http://nbviewer.jupyter.org/github/ibm-cds-labs/spark.samples/blob/master/notebook/DashDB%20Twitter%20Car%202015%20Python%20Notebook.ipynb)\n",
    "____"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spark has pretty well known Streaming Capabilities, if streaming is something you've found yourself needing at work then you are probably familiar with some of these concepts already, in which case you may find it more useful to jump straight to the official documentation here:\n",
    "\n",
    "http://spark.apache.org/docs/latest/streaming-programming-guide.html#spark-streaming-programming-guide\n",
    "\n",
    "It is really a great guide, but keep in mind some of the features are restricted to Scala at this time (Spark 2.1), hopefully they will be expanded to the Python API in the future!\n",
    "\n",
    "For those of you new to Spark Streaming, let's get started with a classic example, streaming Twitter! Twitter is a great source for streaming because its something most people already have an intuitive understanding of, you can visit the site yourself, and a lot of streaming technology has come out of Twitter as a company. You don't access to the entire \"firehose\" of twitter without paying for it, but that would be too much for us to handle anyway, so we'll be more than fine with the freely available API access.\n",
    "\n",
    "_____"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's discuss SparkStreaming!\n",
    "\n",
    "Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Kinesis, or TCP sockets, and can be processed using complex algorithms expressed with high-level functions like map, reduce, join and window. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. In fact, you can apply Spark’s machine learning and graph processing algorithms on data streams.\n",
    "\n",
    "<img src='http://spark.apache.org/docs/latest/img/streaming-arch.png'/>\n",
    "\n",
    "Keep in mind that a few of these Streamiing Capabilities are limited when it comes to Python, you'll need to reference the documentation for the most up to date information. Also the streaming contexts tend to follow more along with the older RDD syntax, so a few things might seem different than what we are used to seeing, keep that in mind, you'll definitely want to have a good understanding of lambda expressions before continuing with this!\n",
    "\n",
    "There are SparkSQL modules for streaming: \n",
    "\n",
    "http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=streaming#module-pyspark.sql.streaming\n",
    "\n",
    "But they are all still listed as experimental, so instead of showing you somethign that might break in the future, we'll stick to the RDD methods (which is what the documentation also currently shows for streaming).\n",
    "\n",
    "Internally, it works as follows. Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.\n",
    "\n",
    "<img src='http://spark.apache.org/docs/latest/img/streaming-flow.png'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Local Example\n",
    "\n",
    "We'll do a simple local counting example, make sure to watch the video for this, the example will only work on Linux type systems, not on a Windows computer. This makes sense because you won't run this on Windows in the real world. Definitely watch the video for this one, a lot of it can't be replicated on Jupyter Notebook by itself!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from pyspark import SparkContext\n",
    "from pyspark.streaming import StreamingContext\n",
    "\n",
    "# Create a local StreamingContext with two working thread and batch interval of 1 second\n",
    "sc = SparkContext(\"local[2]\", \"NetworkWordCount\")\n",
    "ssc = StreamingContext(sc, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a DStream that will connect to hostname:port, like localhost:9999\n",
    "# Firewalls might block this!\n",
    "lines = ssc.socketTextStream(\"localhost\", 9999)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Split each line into words\n",
    "words = lines.flatMap(lambda line: line.split(\" \"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Count each word in each batch\n",
    "pairs = words.map(lambda word: (word, 1))\n",
    "wordCounts = pairs.reduceByKey(lambda x, y: x + y)\n",
    "\n",
    "# Print the first ten elements of each RDD generated in this DStream to the console\n",
    "wordCounts.pprint()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we open up a Unix terminal and type:\n",
    "\n",
    "         $ nc -lk 9999\n",
    "     $ hello world any text you want\n",
    "     \n",
    "With this running run the line below, then type Ctrl+C to terminate it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:46\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:47\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:48\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:49\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:50\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:51\n",
      "-------------------------------------------\n",
      "('world', 1)\n",
      "('hello', 1)\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:52\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:53\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:54\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:55\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:56\n",
      "-------------------------------------------\n",
      "('this', 1)\n",
      "('is', 1)\n",
      "('working', 1)\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:57\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:58\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:31:59\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:00\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:01\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:02\n",
      "-------------------------------------------\n",
      "('i', 1)\n",
      "('check', 1)\n",
      "('need', 1)\n",
      "('to', 1)\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:03\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:04\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:05\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:06\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:07\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:08\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:09\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:10\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:11\n",
      "-------------------------------------------\n",
      "\n",
      "-------------------------------------------\n",
      "Time: 2017-03-29 14:32:12\n",
      "-------------------------------------------\n",
      "\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-e5c44286d287>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mssc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m             \u001b[0;31m# Start the computation\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mssc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mawaitTermination\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# Wait for the computation to terminate\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/Cellar/apache-spark/2.0.1/libexec/python/pyspark/streaming/context.py\u001b[0m in \u001b[0;36mawaitTermination\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    204\u001b[0m         \"\"\"\n\u001b[1;32m    205\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mtimeout\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 206\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jssc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mawaitTermination\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    207\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    208\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_jssc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mawaitTerminationOrTimeout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtimeout\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0;36m1000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/marci/anaconda/lib/python3.5/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m   1129\u001b[0m             \u001b[0mproto\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mEND_COMMAND_PART\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1130\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1131\u001b[0;31m         \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgateway_client\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1132\u001b[0m         return_value = get_return_value(\n\u001b[1;32m   1133\u001b[0m             answer, self.gateway_client, self.target_id, self.name)\n",
      "\u001b[0;32m/Users/marci/anaconda/lib/python3.5/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36msend_command\u001b[0;34m(self, command, retry, binary)\u001b[0m\n\u001b[1;32m    881\u001b[0m         \u001b[0mconnection\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_connection\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    882\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 883\u001b[0;31m             \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconnection\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend_command\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcommand\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    884\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mbinary\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    885\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mresponse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_create_connection_guard\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mconnection\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/marci/anaconda/lib/python3.5/site-packages/py4j/java_gateway.py\u001b[0m in \u001b[0;36msend_command\u001b[0;34m(self, command)\u001b[0m\n\u001b[1;32m   1026\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1027\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1028\u001b[0;31m             \u001b[0manswer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msmart_decode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstream\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1029\u001b[0m             \u001b[0mlogger\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdebug\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Answer received: {0}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0manswer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1030\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0manswer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mproto\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRETURN_MESSAGE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/marci/anaconda/lib/python3.5/socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    574\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    575\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 576\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    577\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    578\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "ssc.start()             # Start the computation\n",
    "ssc.awaitTermination()  # Wait for the computation to terminate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Twitter Example\n",
    "In order to use all of this though, we need to setup a Developer API acocunt with Twitter and create an application to get credentials. Review the video for instructions on how to do this or if you are already familiar with it, just get the credentials from: \n",
    "\n",
    "    https://apps.twitter.com/\n",
    "    \n",
    "Once you have that you also need to install python-twitter, a python library to connect your Python to the twitter dev account.\n",
    "\n",
    "You probably won't be able to run this example and then previous in the same notebook, you need to restart you kernel.\n",
    "\n",
    "Let's get started!\n",
    "\n",
    "Begin by running the TweetRead.py file. Make sure to add your own IP Adress and your credential keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# May cause deprecation warnings, safe to ignore, they aren't errors\n",
    "from pyspark import SparkContext\n",
    "from pyspark.streaming import StreamingContext\n",
    "from pyspark.sql import SQLContext\n",
    "from pyspark.sql.functions import desc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Can only run this once. restart your kernel for any errors.\n",
    "sc = SparkContext()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ssc = StreamingContext(sc, 10 )\n",
    "sqlContext = SQLContext(sc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "socket_stream = ssc.socketTextStream(\"127.0.0.1\", 5555)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lines = socket_stream.window( 20 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "fields = (\"tag\", \"count\" )\n",
    "Tweet = namedtuple( 'Tweet', fields )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Use Parenthesis for multiple lines or use \\.\n",
    "( lines.flatMap( lambda text: text.split( \" \" ) ) #Splits to a list\n",
    "  .filter( lambda word: word.lower().startswith(\"#\") ) # Checks for hashtag calls\n",
    "  .map( lambda word: ( word.lower(), 1 ) ) # Lower cases the word\n",
    "  .reduceByKey( lambda a, b: a + b ) # Reduces\n",
    "  .map( lambda rec: Tweet( rec[0], rec[1] ) ) # Stores in a Tweet Object\n",
    "  .foreachRDD( lambda rdd: rdd.toDF().sort( desc(\"count\") ) # Sorts Them in a DF\n",
    "  .limit(10).registerTempTable(\"tweets\") ) ) # Registers to a table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__________\n",
    "### Now run TweetRead.py\n",
    "__________"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ssc.start()    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import time\n",
    "from IPython import display\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "# Only works for Jupyter Notebooks!\n",
    "%matplotlib inline "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAApcAAAHfCAYAAAAFj/MNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2YXWV57/HvhkGhMBaQjUGPGk6pNyDISyyGl0BAqWK1\ngCVWEDQgIoICWlu8lGLa8nqkilblHOMLVWsLIQWEWj05IAREJRoTiA13S7SAWnCCIOGIlZDdP9bK\nYTOHSYb4zF6z93w/15Vr9l7rWeu59zzXJL/ca+3ZrU6ngyRJklTCZk0XIEmSpMFhuJQkSVIxhktJ\nkiQVY7iUJElSMYZLSZIkFWO4lCRJUjFDTRegytq1T3QeeuiXTZehjdhuu9/CdZr8XKf+4Vr1B9ep\nP/Ryndrt4dZY++xcThJDQ5s3XYLGwXXqD65T/3Ct+oPr1B8myzoZLiVJklSM4VKSJEnFeM/lJDFy\n2ZeaLkHjMNJ0ARoX16l/uFb9wXXqE+88vukKADuXkiRJKshwKUmSpGIMl5IkSSrGcClJkqRiDJeS\nJEkqxnApSZKkYgyXkiRJKsZwKUmSpGIMl5IkSSrGcClJkqRiDJeSJEkqxnApSZKkYgyXkiRJKqbx\ncBkR0yLi8vrxogmea25EXFQ/vjgi7oiI2RM5pyRJ0lQy1HQBwExgSURsDTzaw3nnAHtl5poezilJ\nkjTQGg2XEbEAOBC4FzgNGIqIFcCtwK5AC/jj+vHFwK+BTwP3A+cBvwIeBE4C9h415l7gfOAJYBXw\njq55zwWeD/xTRLy6PtdB9e4vZ+bH6m7qc+s/Hwb+ODPfVB9/f2ZOi4g3AGcDjwM/Bd5Un/cyYEtg\nJ+CczLym2DdNkiRpEmv0snhmzgGWAbOAK4CTgdXAbZk5u972gXr4lpk5C/gSVXh8Q2YeAtwMnPM0\nY+Z3jfkJMLdr3r+kCqi/D7wS2Jmqg3oQcFxE7FkPvTEzDwAeGuMlHAt8ODMPAq4HnkMVhP86Mw8H\nTgFO36RvjiRJUh9qNFxGxEJgP2ARcAZwAbADcGM95DYg6sdZf90BeCQzf1I/Xwy8dNSYNlXX8MqI\nuIkqRL54jDJ2A27JzE5mPg58G9h91PlGa9Vf3wscFhE3AwcA64D/AN4REV8ETgW2GOv1S5IkDZqm\n39BzITC/7lIur7uOq4EZ9f4DgR/Uj9fVX1cDz4mInernhwD/+jRjfgwcWZ/7fJ4MrKOtpL4kHhFb\nUIXEfxt1vl9RhVUi4sXA9vX2U4B5dXe0BRwN/BXwhcw8AfgGTwZRSZKkgdd0uJwBLI2I6VT3Ra43\nt+4G/gFVMPx/MrMDvB34x4j4JvAqqkDXPWYdcCbVPZW3Ud3PueLpCsjM64EfRcS3qLqWV2Xm0lHD\nvgs8HBHfAf4C+FG9/Xbg+oi4AZhGdWl8AXBJRCwGDqfqtEqSJE0JrU6n03QNT1Ffxj41M+9qupZe\nGrnsS5NrISRJUl9pv/N4RkZ680tw2u3hMa/MNt25lCRJ0gCZDL/n8inqeyQlSZLUh+xcSpIkqRjD\npSRJkooxXEqSJKkYw6UkSZKKMVxKkiSpGMOlJEmSijFcSpIkqRjDpSRJkooxXEqSJKkYw6UkSZKK\nMVxKkiSpmFan02m6BlU6IyNrmq5BG9FuD+M6TX6uU/9wrfqD69QferlO7fZwa6x9di4lSZJUjOFS\nkiRJxRguJUmSVIzhUpIkScUYLiVJklSM4VKSJEnFGC4lSZJUjOFSkiRJxQw1XYAqd33yyKZL0Dg8\n2HQBGhfXqX+4Vv3BdeoP7dOvbboEwM6lJEmSCjJcSpIkqRjDpSRJkooxXEqSJKkYw6UkSZKKMVxK\nkiSpGMOlJEmSijFcSpIkqRjDpSRJkooxXEqSJKkYw6UkSZKKMVxKkiSpGMOlJEmSipnU4TIipkXE\n5fXjRRFxU0TsWvD8346I6aXOJ0mSNNVN6nAJzASWRMTWwKNNFyNJkqQNG2q6gLFExALgQOBe4DSq\nWh8HPhQRzwO2Bo7NzB9GxIXALGBz4COZuSAibgKWAXsAzwHmZOY9EXE+8BrgPmCHeq7fBj4LPLee\n/ozMvDMi7gHuAv4FuAU4u67hp8Cb6vN+qf46BJyTmTdGxB3AzcDLgA5wZGb+YmK+U5IkSZPHpO1c\nZuYcqnA4C7gCOBlYDfxTZh4G/DNwTEQcAeycmQcBhwIfjIht69PcnpmvAhYBx0bEy4GDgd8D3gIM\n1+M+ANyQmYcCpwCX1dtfCByXme8BjgU+XM9zPVWgPAdYlJkHA3OAz0ZEq97395l5CPAT4Ijy3yFJ\nkqTJZ9KGy4hYCOxHFQzPAC6g6jR+rx5yP/BbwJ7AjLpT+TVgC2B6Peb79df7gC2BlwDfzcx1mfkI\ncGe9f0/gpPoc84Ht6+2rM/PB+vF7gcMi4mbgAGAdsBuwGCAzfwI8Auw4xtySJEkDb9KGS+BCYH5m\nzgaWZ+Ysqs5lZ9S4u4Bv1OMOA64EVtX7Ro/9F2C/iNisvo9z965zfLQ+xxupLnVDFSDXOwWYV3cj\nW8DRwEqqzioR8QJgO2B9GB09tyRJ0sCbzOFyBrC0fjf3qg2Muw54NCJuoepqdjJzzdMNzMxlVJfT\nlwD/APys3nU+8Mau7ueKpzn8duD6iLgBmEZ1afwCqm7mYuAa4JTMXPtMXqQkSdIgaXU6Ntgmg7s+\neaQLIUmSNtmup1/LyMjT9teKa7eHW2Ptm8ydS0mSJPUZw6UkSZKKMVxKkiSpGMOlJEmSijFcSpIk\nqRjDpSRJkooxXEqSJKkYw6UkSZKKMVxKkiSpGMOlJEmSijFcSpIkqRjDpSRJkooxXEqSJKmYVqfT\naboGVTojI2uarkEb0W4P4zpNfq5T/3Ct+oPr1B96uU7t9nBrrH12LiVJklSM4VKSJEnFGC4lSZJU\njOFSkiRJxRguJUmSVIzhUpIkScUYLiVJklTMUNMFqHLV51/TdAmSJKmPHXPi15ouAbBzKUmSpIIM\nl5IkSSrGcClJkqRiDJeSJEkqxnApSZKkYgyXkiRJKsZwKUmSpGIMl5IkSSrGcClJkqRiDJeSJEkq\nxnApSZKkYgyXkiRJKsZwKUmSpGKGejVRREwDLsrMuRGxCJgDnDVq2MOZeWk9/ihg71H7r8nMZRGx\nbb8diyRJ0hTQs3AJzASWRMTWwKPAtsB5mbl2/YCI2KVr/IrMvKb7BF37+/FYSZKkgdeTy+IRsQD4\nFHACcDuwO3BML+aWJElS7/QkXGbmHGAZMAu4AjgZuKoXc0uSJKl3enJZPCIWAvsBi4A9gMOBa4FL\nezG/JEmSeqNX7xa/EJifmbOB5Zk5CzuXkiRJA6dXb+iZASyNiOnAqq7t50bEuq7nmwHn1o8Pjojj\nR51nMXB3Hx8rSZI00FqdTqfpGgRc9fnXuBCSJGmTHXPi1xgZWdOTudrt4dZY+/wl6pIkSSrGcClJ\nkqRiDJeSJEkqxnApSZKkYgyXkiRJKsZwKUmSpGIMl5IkSSrGcClJkqRiDJeSJEkqxnApSZKkYgyX\nkiRJKsZwKUmSpGIMl5IkSSqm1el0mq5Blc7IyJqma9BGtNvDuE6Tn+vUP1yr/uA69YderlO7Pdwa\na5+dS0mSJBVjuJQkSVIxhktJkiQVY7iUJElSMYZLSZIkFWO4lCRJUjGGS0mSJBVjuJQkSVIxQ00X\noMq8K1/ddAmSJKmPzXvj15suAbBzKUmSpIIMl5IkSSrGcClJkqRiDJeSJEkqxnApSZKkYgyXkiRJ\nKsZwKUmSpGIMl5IkSSrGcClJkqRiDJeSJEkqxnApSZKkYgyXkiRJKsZwKUmSpGKmRLiMiGkRcXn9\neFHD5UiSJA2sKREugZnAkojYGni06WIkSZIG1VDTBUy0iFgAHAjcC5wGDEXEucAfAOuAJZl5RkRM\nBz5H9T3pAGdk5vKIeBvwTmBz4CuZ+aExts0B3gs8Adyame+PiHnAAcA2wNsyc2XPXrgkSVIDBr5z\nmZlzgGXALOAK4GTg9cC7MnN/YGVEDAGXAB/LzIOBM4HPRsSOwPvrY/cFnh0RLxpj218Ar8zMg4AX\nRMThdQkrM/MAg6UkSZoKpkLnciGwH7AI2AM4HLgWOD0idga+BbSA3YDFAJm5LCJeCPx3YEVmPlaf\n7v0RMfNptu0HtIGvRgTAMPA79f6c4JcoSZI0aQx85xK4EJifmbOB5Zk5iyoInpqZhwD7UF26XknV\njSQi9gbuB1YBu0bEs+vtVwH/8TTbHgDuAw6v5/kb4Nv1/Ot68BolSZImhakQLmcAS+t7KlfV2+4E\nbomIG4GfAd8B3ge8OyIWA5dR3SM5AlwM3BwR3wKWZuY9Y2z7SL3tO8ARwL/27BVKkiRNEq1Op9N0\nDQLmXflqF0KSJG2yeW/8OiMja3oyV7s93Bpr31ToXEqSJKlHDJeSJEkqxnApSZKkYgyXkiRJKsZw\nKUmSpGIMl5IkSSrGcClJkqRiDJeSJEkqxnApSZKkYgyXkiRJKsZwKUmSpGIMl5IkSSqm1el0mq5B\nlU6vPmxem67dHsZ1mvxcp/7hWvUH16k/9HKd2u3h1lj77FxKkiSpGMOlJEmSijFcSpIkqRjDpSRJ\nkooxXEqSJKkYw6UkSZKKMVxKkiSpGMOlJEmSihlqugBVXnv1eU2XIEmS+thXjz6n6RIAO5eSJEkq\nyHApSZKkYgyXkiRJKsZwKUmSpGIMl5IkSSrGcClJkqRiDJeSJEkqxnApSZKkYgyXkiRJKsZwKUmS\npGIMl5IkSSrGcClJkqRiDJeSJEkqZqjpAiZSREwDLsrMuRGxCPgB8JHMvDcizgK2HXXIpZn5cETs\nDRw1at+yzLymPu+Yx46a/0XAXpl5XanXJEmSNJkNdLgEZgJLImJr4NHMPKtr3/WZeff6JxExBPw3\n4OF67LzuE0XELuM8ttthwK6A4VKSJE0JAxsuI2IBcCBwL3AaMBQRK4BjMvOuCZz3HuAu4F+AI4Df\niojbMvMrEzWnJEnSZDGw91xm5hxgGTALuAI4GVjdg6lfCByXme8BLgK+bLCUJElTxSB3LhcC+wGL\ngD2Aw4HtejD16sx8sAfzSJIkTToD27kELgTmZ+ZsYHlmzqI3nct1ox4P8vdYkiTpKQa2cwnMAJZG\nxHRg1dPsf0tEdAfBzYDP1Y9fFBHHjxp/L7D+TTwbOrbbncAHI2JpZv7DM30BkiRJ/abV6XSarqFn\nIuKbwAmZ+cOmaxnttVefN3UWQpIkFffVo89hZGRNT+Zqt4dbY+2bMpdsI+LjVJep72m6FkmSpEE1\nyJfFnyIzz2i6BkmSpEE3ZTqXkiRJmniGS0mSJBVjuJQkSVIxhktJkiQVY7iUJElSMYZLSZIkFWO4\nlCRJUjGGS0mSJBVjuJQkSVIxhktJkiQVs9GPf4yIzwOdrk0d4DFgJTA/M389QbVJkiSpz4zns8XX\nAtsDf1s/fxMwDDwB/E/gpIkpbWr56tHnMDKypukytBHt9rDr1Adcp/7hWvUH10nPxHjC5T6Z+fL1\nTyLiOuA7mfnGiFg+caVJkiSp34znnsutI2Ja1/Mdga3qx+MJp5IkSZoixhMOPwR8LyJuAzYHXg6c\nGRHzgEUTWJskSZL6zEbDZWZeGRE3ArOo7rM8JTNXR8TNmfnzCa9QkiRJfWM87xbfEXgzsA3QAmZE\nxM6Z+ZaJLk6SJEn9ZTz3XP4jsDdwPLA18IfAuoksSpIkSf1pPOFyh8x8K3AdVdCcDbx0IouSJElS\nfxpPuHyo/prAXpn5C+BZE1eSJEmS+tV43i1+Q0QsAN4H/O+I2Bf45cSWNfW87qq/a7oESZLUx64/\n5s1NlwCMr3O5G3B2Zt4DHEvVwTRcSpIk6f8zZucyIq4G9gKeD+wTEet3bQHcM/GlSZIkqd9s6LL4\nW6k+U/xjwBld29cCD0xkUZIkSepPY4bLzHwEeAQ4snflSJIkqZ+N555LSZIkaVwMl5IkSSrGcClJ\nkqRiDJeSJEkqxnApSZKkYgyXkiRJKsZwKUmSpGIMl5IkSSrGcClJkqRipmS4jIhpEXF5/XjRGGMu\njYgXFZjrpojY9Tc9jyRJUj/Y0GeLD7KZwJKI2Bp49OkGZOZZvS1JkiSp/025cBkRC4ADgXuB04At\nI2IlsHtmdiLiE8ANwJnAqcBzgb8GHgd+CRwD7AR8HlhL1f09LjPvi4gLgVnA5sBHMnNBT1+cJElS\nw6bcZfHMnAMsowqBVwBzgTuAWRHxbOBQ4LquQ44CrgQOAS4DtgMOB24HXgV8CPjtiDgC2DkzD6rP\n8cGI2LYXr0mSJGmymHLhMiIWAvsBi4AzgAuABcBbgSOBr2Tm2q5DLgCeT9XNPIaqg/lZ4GHga8C7\nqDqYewIzIuKmevsWwPQJf0GSJEmTyJQLl8CFwPzMnA0sz8xZwEJgH+Ak4DOjxh8PXJ6ZhwI/AE6h\nCqG3ZOYrqYLp2cBdwDfq8x5G1e1cNeGvRpIkaRKZiuFyBrA0IqZTh7/M7ABXAc/KzNGB8HbgMxFx\nA1Vo/ALwXeAvI+JGqvsy/4bqUvqjEXEL8D2gk5lrevB6JEmSJo1Wp9NpugYBr7vq71wISZK0ya4/\n5s2MjPSmr9VuD7fG2jcVO5eSJEmaIIZLSZIkFWO4lCRJUjGGS0mSJBVjuJQkSVIxhktJkiQVY7iU\nJElSMYZLSZIkFWO4lCRJUjGGS0mSJBVjuJQkSVIxhktJkiQV0+p0Ok3XoEqnVx82r03Xbg/jOk1+\nrlP/cK36g+vUH3q5Tu32cGusfXYuJUmSVIzhUpIkScUYLiVJklSM4VKSJEnFGC4lSZJUjOFSkiRJ\nxRguJUmSVIzhUpIkScUMNV2AKkcvvLXpEiRJUh+7+o8OaroEwM6lJEmSCjJcSpIkqRjDpSRJkoox\nXEqSJKkYw6UkSZKKMVxKkiSpGMOlJEmSijFcSpIkqRjDpSRJkooxXEqSJKkYw6UkSZKKMVxKkiSp\nGMOlJEmSihlquoCmRcQ04KLMnBsRi4A5wFmjhj2cmZfW448C9h61/5rMXBYR227oWEmSpEE35cMl\nMBNYEhFbA48C2wLnZeba9QMiYpeu8Ssy85ruE3Tt39ixkiRJA21KXxaPiAXAp4ATgNuB3YFjGi1K\nkiSpj03pcJmZc4BlwCzgCuBk4KpGi5IkSepjU/qyeEQsBPYDFgF7AIcD1wLeIylJkrQJpnTnErgQ\nmJ+Zs4HlmTkLO5eSJEmbbEp3LoEZwNKImA6s6tp+bkSs63q+GXBu/fjgiDh+1HkWA3eP41hJkqSB\n1up0Ok3XIODohbe6EJIkaZNd/UcHMTKypidztdvDrbH2TfXL4pIkSSrIcClJkqRiDJeSJEkqxnAp\nSZKkYgyXkiRJKsZwKUmSpGIMl5IkSSrGcClJkqRiDJeSJEkqxnApSZKkYgyXkiRJKsZwKUmSpGIM\nl5IkSSqm1el0mq5Blc7IyJqma9BGtNvDuE6Tn+vUP1yr/uA69YderlO7Pdwaa5+dS0mSJBVjuJQk\nSVIxhktJkiQVY7iUJElSMYZLSZIkFWO4lCRJUjGGS0mSJBVjuJQkSVIxQ00XoMonr36g6RI0Lr9s\nugCNi+vUP1yr/uA69YPTjx5uugTAzqUkSZIKMlxKkiSpGMOlJEmSijFcSpIkqRjDpSRJkooxXEqS\nJKkYw6UkSZKKMVxKkiSpGMOlJEmSijFcSpIkqRjDpSRJkooxXEqSJKkYw6UkSZKKmbBwGRHTIuLy\n+vGiiZqnPv8/PoOx/x4RW05kPZIkSVPVRHYuZwJLImJr4NEJnIfMfMNEnl+SJEnjMzQRJ42IBcCB\nwL3AacBQRKwAbgV2BVrAH9ePLwZ+DXy6Hn8+8ASwCngH8Gbg9cBWwE7Ax4AjgT2A92XmtRFxf2ZO\ni4g9gY/X538QOAnYZ9Qc62t8Yf18K+Ax4JTMvC8iLgReDjwXWJ6ZJ0bEDsCXgWcDCRyWmbtExL8D\nu2bmryLiIuCuzLy8PscsYHPgI5m5oNg3V5IkaRKbkM5lZs4BllEFrCuAk4HVwG2ZObve9oF6+JaZ\nOQv4EjAfeENmHgL8BJhbjxnOzNdShcR3Am8ATgFOHDX1fOD0eo6vAn/WPUdmfrFr7CXAx+uxlwAX\nRcRzgIcy83CqgDkzIl4AfBC4pq5rARsI5RFxBLBzZh4EHAp8MCK23eg3TZIkaQBMVOdyIbAfsIiq\nw3g4sB1wYz3kNqruI1SdQIA2VWfyyoiAqqO4CLgb+H495mFgZWZ2IuIhYPS9k7sBn6qP3wL4t1Fz\ndNsT+EBEnE3V6XycqoO5Y0T8PdWl/G3q8+wG/G193C1jvOxW13lnRMRN9fMtgOlUYVuSJGmgTdQ9\nlxcC8+uu4PK6M7kamFHvPxD4Qf14Xf11NfBj4Mj6uPN5Mox2xjlvAm+pj/8z4PpRc3S7Czi7HvsO\nqo7kEcALM/NYqs7qVlShcQWwf33czK5z/ArYKSJawN5d5/1Gfd7DgCupLvFLkiQNvAnpXFKFyKUR\nMZ2nBqu5EfFe4P8CJ1B1+QDIzHURcSbwTxGxGfAI8BbgRc9g3ncCX4iIIapA+jbg+WOMfR9wWf3O\n8a2AM4EfAX8eEYvr439YH38R8MWIeCPwU6ouJ8D/oLr8/u/AQ/W264DZEXELVefz6sxc8wxegyRJ\nUt9qdTrjbQr+ZurLxKdm5l09mbCgiHgtMJKZSyLiVcAHMvOwknN88uoHerMQkiRpIJ1+9PMYGelN\nP6vdHm6NtW+iOpeD5kfA5yJiLdU7wM9ouB5JkqRJqWfhsr4HsS9l5kqevOdSkiRJY/DjHyVJklSM\n4VKSJEnFGC4lSZJUjOFSkiRJxRguJUmSVIzhUpIkScUYLiVJklSM4VKSJEnFGC4lSZJUjOFSkiRJ\nxbQ6nU7TNajS6dWHzWvTtdvDuE6Tn+vUP1yr/uA69YderlO7Pdwaa5+dS0mSJBVjuJQkSVIxhktJ\nkiQVY7iUJElSMYZLSZIkFWO4lCRJUjGGS0mSJBVjuJQkSVIxQ00XoMr3P/OzpkvQOPyYx5ouQePg\nOvUP16o/uE79oX3ycNMlAHYuJUmSVJDhUpIkScUYLiVJklSM4VKSJEnFGC4lSZJUjOFSkiRJxRgu\nJUmSVIzhUpIkScUYLiVJklSM4VKSJEnFGC4lSZJUjOFSkiRJxRguJUmSVMxQ0wVsqoiYBlyUmXMj\nYhEwBzhr1LCHM/PSevxRwN6j9l+Tmcue4bzzRm/LzHn1vtnA7FG7b8rMm57JHJIkSf2qb8MlMBNY\nEhFbA48C2wLnZeba9QMiYpeu8Ssy85ruE4zaP15fysy7xzjHj9cHzd9wDkmSpL7Ul+EyIhYABwL3\nAqdRvY5jgEs34Vw7ALcAu2dmJyI+AdwA7AS8FVgHLMnMMwqVL0mSNLD68p7LzJwDLANmAVcAJwNX\nbeK5VgN3ALMi4tnAocB1wInAuzJzf2BlRPRlEJckSeqlvgyXEbEQ2A9YBJwBXEDVudxU86m6lEcC\nX6kvrZ8InB4RNwMvBlq/UdGSJElTQF+GS+BCYH5mzgaWZ+YsNrFzWbsB2Ac4CfhMve3twKmZeUi9\n74Df4PySJElTQr9e6p0BLI2I6cCqru3nRsS6ruebAefWjw+OiONHnWcxcHd9r+VVwKsyc/357gRu\niYg1wE+A79Tbz4iIn3edY3uq7inAy55mjjuAu5EkSZoCWp1Op+kaBHz/Mz9zISRJ0ibb5+QdGRlZ\n05O52u3hMW8X7NfL4pIkSZqEDJeSJEkqxnApSZKkYgyXkiRJKsZwKUmSpGIMl5IkSSrGcClJkqRi\nDJeSJEkqxnApSZKkYgyXkiRJKsZwKUmSpGIMl5IkSSrGcClJkqRiWp1Op+kaVOmMjKxpugZtRLs9\njOs0+blO/cO16g+uU3/o5Tq128OtsfbZuZQkSVIxhktJkiQVY7iUJElSMYZLSZIkFWO4lCRJUjGG\nS0mSJBVjuJQkSVIxhktJkiQVM9R0Aao8cOn3mi5B4/BA0wVoXFyn/uFa9QfXqU+cNaPpCgA7l5Ik\nSSrIcClJkqRiDJeSJEkqxnApSZKkYgyXkiRJKsZwKUmSpGIMl5IkSSrGcClJkqRiDJeSJEkqxnAp\nSZKkYgyXkiRJKsZwKUmSpGIMl5IkSSpmqOkCJlpETAMuysy5EbEImAOcNWrYw5l5aT3+KGDvUfuv\nycxlEbHtph5b4rVIkiRNdgMfLoGZwJKI2Bp4FNgWOC8z164fEBG7dI1fkZnXdJ+ga/9vcqwkSdLA\nG+jL4hGxAPgUcAJwO7A7cEyjRUmSJA2wgQ6XmTkHWAbMAq4ATgauarQoSZKkATbQl8UjYiGwH7AI\n2AM4HLgWuLTJuiRJkgbVQHcugQuB+Zk5G1iembOwcylJkjRhBrpzCcwAlkbEdGBV1/ZzI2Jd1/PN\ngHPrxwdHxPGjzrMYuLvAsZIkSQOt1el0mq5BwAOXfs+FkCRJm+x5Z81gZGRNT+Zqt4dbY+0b9Mvi\nkiRJ6iHDpSRJkooxXEqSJKkYw6UkSZKKMVxKkiSpGMOlJEmSijFcSpIkqRjDpSRJkooxXEqSJKkY\nw6UkSZKKMVxKkiSpGMOlJEmSiml1Op2ma1Cl06sPm9ema7eHcZ0mP9epf7hW/cF16g+9XKd2e7g1\n1j47l5IkSSrGcClJkqRiDJeSJEkqxnApSZKkYnxDjyRJkoqxcylJkqRiDJeSJEkqxnApSZKkYgyX\nkiRJKsZwKUmSpGIMl5IkSSpmqOkCppKI2Az4FLAX8J/AyZl5d9f+1wPnAmuBz2Xm/EYK1XjW6ljg\nLKq1uhM4LTPXNVHrVLaxdeoa92ng55n5/h6XKMb18/R7wEeAFnA/cHxm/qqJWqeycazTm4E/AZ6g\n+jfqskYKFQAR8Qrg4sycPWp741nCzmVvHQVsmZn7A+8H/nr9jojYAvgo8PvAIcApEfG8RqoUbHit\ntgLOAw7NzAOB3wZe10iVGnOd1ouIdwB79rowPcWGfp5awHzgxMw8CPga8OJGqtTGfp4uAV4FHAj8\nSURs1+P6VIuIPwM+A2w5avukyBKGy95a/xcnmflt4OVd+3YD7s7MhzLz18CtwMG9L1G1Da3VfwIH\nZOYv6+enXbcCAAAFLElEQVRDgF2WZmxonYiIA4BXAP+r96Wpy4bW6SXAg8B7IuJmYPvMzN6XKDby\n8wTcQfWf6S2pusx+CktzVgFveJrtkyJLGC576znAL7qePxERQ2PsW0P1Q6xmjLlWmbkuMx8AiIh3\nA9sAi3pfotjAOkXETsCHgHc1UZieYkN/9+0AHAB8gqor9sqIOKzH9amyoXUCWAF8D/gBcH1mPtzL\n4vSkzFwIPP40uyZFljBc9tYjwHDX880yc+0Y+4YBf3Cbs6G1IiI2i4hLgMOBP8pM/wffjA2t0xyq\n4PJVqkt8x0XE3N6Wp9qG1ulBqk7Lysx8nKpzNrpjpt4Yc50i4mXAHwA7A9OBHSNiTs8r1MZMiixh\nuOytbwKvBYiImVRvBFlvJfC7EbF9RDyLqo39rd6XqNqG1gqqy6xbAkd1XR5X7425Tpn58cycUd/s\nfhHw5cy8vIkitcGfpx8C20TELvXzWVSdMfXehtbpF8BjwGOZ+QTwM8B7LiefSZElWp2ODZde6Xon\n3suo7lc5EdgX2CYzP931Dq/NqN7h9cnGip3iNrRWwHfrP7fw5D1HH8vMqxsodUrb2M9U17i5wK6+\nW7wZ4/i77zCq/wC0gNsy88zGip3CxrFOpwInAb+muufv7fV9fWpAREwH/iEzZ0bEcUyiLGG4lCRJ\nUjFeFpckSVIxhktJkiQVY7iUJElSMYZLSZIkFWO4lCRJUjGGS0kaQBGxb0Rc3KO5vlF/3Swiro6I\nbXoxr6TJyXApSYPpo0BPwiUwG6qPRgXmU/2OPUlTlL/nUpJ6ICJmAx+k+uXUvwNcRfWpJ0fV214L\n7AP8JbAF8COqX1L9YP0xe38CbFX/OTkzF0fETcDtVJ9q0wbenZn/XP9S8lMy80313McB51D90v8l\nwNvrOeYDewHrgEsy8wv1L5yfnZlz62NvAubVL+MDwC+B3ag+veU44BLg3cDtmfmKiNgcSGDfzHyk\n2DdQUt+wcylJvfMKqk89eSnwTmAkM18O3AGcSvUpNa/OzH2ArwMX15+acirwuszcqx7zp13nfFZm\n7g+8Bziv3vaHwGKAiHgBVRfz9zPzpcDmVJ8RPQ94MDP3AA4D5tWfH70hBwDvogqXL6prPQMgM19R\nf32ifj2HPuPvjqSBMNR0AZI0hazIzPsAImI1cEO9/R7g9VSB7RsRAVUI/HlmrouIo4HXR7VjNvBE\n1zm/tv7cwPb1498Fbqwf7w98MzN/DJCZJ9TznwO8rd62OiKurc+9oW7jivXniYiVXfONdk9dg6Qp\nyHApSb0z+nOY13Y93hy4NTP/ECAitgSG6zfHLAG+SNWNvIOqe7jer+qvHarL61Bd5l5/7se7J4yI\ndv1w9JWrFtW/Cd3ngery+ei5Rs832uN1DZKmIC+LS9Lk8B1g/4h4Sf38z4EPAy+hCmoXUHUjj6AK\nohuyCnhx/XgJ8IqImFY//yhwZH2utwFExA5U937eBKwGdouIVkTsDGzsUjnAExHR3azYGbh7HMdJ\nGkCGS0maHO4HTgKujIg7gX2p3sSzHFgG3AUsBR7lyeA4luuo73nMzJ8CZwJfj4gVwGPA56neOLR9\nPddi4PzMXAr8H+A+qjflfAy4dRy1Xwssj4gt6zf07FufR9IU5LvFJWnARESLKhQemZmrezz3kcBB\nmfmnGx0saSDZuZSkAZOZHeAs4Oxezlu/s/1twF/1cl5Jk4udS0mSJBVj51KSJEnFGC4lSZJUjOFS\nkiRJxRguJUmSVIzhUpIkScUYLiVJklTMfwEROO+hIv+IsQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11b6cf160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "count = 0\n",
    "while count < 10:\n",
    "    \n",
    "    time.sleep( 3 )\n",
    "    top_10_tweets = sqlContext.sql( 'Select tag, count from tweets' )\n",
    "    top_10_df = top_10_tweets.toPandas()\n",
    "    display.clear_output(wait=True)\n",
    "    sns.plt.figure( figsize = ( 10, 8 ) )\n",
    "    sns.barplot( x=\"count\", y=\"tag\", data=top_10_df)\n",
    "    sns.plt.show()\n",
    "    count = count + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ssc.stop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
